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Abstract 

Correlational analysis of two years of classroom observation indicates rela¬ 
tionships between technology use and various classroom characteristics, in¬ 
cluding teacher roles and instructional strategies. Three observers used the 
ISTE Classroom Observation Tool (ICOT) to record 144 observations of class¬ 
rooms participating in a variety of educational technology grant programs 
in the United States. The findings suggest that decision makers consider the 
instructional implications and possible unintended consequences of imple¬ 
menting different information and communication technologies. (Keywords: 
ISTE Classroom Observation Tool, ICOT, observation, planning, classroom, 
evaluation) 


C lassroom observation plays a crucial role in program evaluation 
(Worthen, Sanders, & Fitzpatrick, 1997). Schools can count up 
technology units installed and hours of professional development as 
inputs to a program. Surveys and interviews can record student and teacher 
attitudes about an innovation. Tests and school transcripts can quantify out¬ 
comes. But only observation of students and teachers at work can document 
the learning experience itself. In particular, observation helps determine 
whether an intervention has actually been implemented before a program 
tries to evaluate outcomes (Frechtling, 2002). Observation thus fills in the 
information required by the middle stages of intervention logic models (see 
Figure 1, p. 206). Without visible evidence that new practices are in effect 
(stage 3 of the model), studies of student outcomes (stage 4) that are sup¬ 
posed to be engendered by those practices may be moot. 

Since 1999, the International Society for Technology in Education 
(ISTE) has conducted classroom observations as part of its evaluations of 
numerous initiatives funded by federal, state, and private grants (Biele¬ 
feldt, 2000, 2003; Kelly & Haber, 2003). In these projects, the point of the 
observations was to determine whether information and communications 
technologies (ICT) were being integrated into instruction. This is more 
complicated than noting that technologies are present and that students 
are attending to them. Numerous attributes determine what constitutes 

Volume 44 Number 3 | Journal of Research on Technology in Education | 205 
Copyright © 2012, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int'l), iste@iste.org, iste.org. All rights reserved. 



Bielefeldt 



Evaluated by records, Evaluated by Evaluated by Evaluated by individual 

schedules, rosters, surveys and classroom products assessments within a 
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Figure 1. Classroom observations within an educational technology logic model. 


true integration as defined by the National Educational Technology 
Standards (NETS), including alignment with curriculum and instruction, 
active interaction with technology tools, and the use of technology to pro¬ 
mote a range of important cognitive skills (ISTE, 2000, 2002, 2007, 2008). 
Making this determination is important for two reasons. First, ICT inte¬ 
gration can be a goal in itself to make school learning environments more 
like postsecondary education, workplaces, and homes in terms of technol¬ 
ogy access and use. Second, assessing outcomes related to ICT integration 
(e.g., student achievement or attitudes) is moot unless a project can show 
that classrooms actually changed. 

In 2008, the Hewlett Packard Company funded ISTE to develop a 
computer-based note-taking application that would take advantage of tablet 
(pen-based) technology. The program, the ISTE Classroom Observation 
Tool (ICOT), can be used on any Windows or Macintosh computer and is 
optimized for tablets, which allow the user to hold the computer in one arm 
and write on the screen with a stylus. ICOT was based on protocols em¬ 
ployed in previous ISTE program evaluations, particularly those presented 
in Kelly (2003); Kelly & Haber (2006). 

Over the years that led up to ICOT, it became clear that the NETS are 
necessary but not sufficient for the purpose of program evaluation and 
improvement. The standards themselves do not describe how they are to 
be achieved, but that is the point of most project evaluations. Although 
program evaluations may not have the theory-building imperative of pure 
research, they are generally charged with estimating whether and how a suc¬ 
cessful initiative could be replicated or scaled. ISTE makes clear that achiev¬ 
ing the standards is contingent on a number of essential conditions (ISTE, 
2000, 2008). The ICOT looks at seven attributes of the learning environment 
that (a) are related by theory or experience to ICT integration and that (b) 
can be noted by classroom observation: 

• Student groupings (individual, pairs/small groups, whole class) 

• Teacher roles (lecture, model, interactive direction, moderation, facilitation) 

• Learning activities (a lengthy list, with space for additions) 

• Technologies used by teachers and by students (a lengthy list, with space 

for additions) 
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• Technology use time (recorded as presence during 3-minute segments of 
the class period) 

• Percent of students engaged (estimated by noting students distracted dur¬ 
ing any two 3-minute segments) 

Some important attributes, such as levels of teacher preparation or tech¬ 
nology support, are essential conditions that are not explicitly included in 
the ICOT because they are not directly observable. Observers can comment 
on the extent to which these conditions are implied, but the circumstances 
under which they might become manifest are too diverse to code in a proto¬ 
col that is already attending to numerous other aspects of the environment. 

Explaining Variation within Projects 

The present study arose out of questions related to the ISTE evaluation of 
several educational technology grant programs from 2008-2010. Although 
the projects varied in specific focus, grade level, and technologies used, certain 
common emphases appear in the projects’ grant announcements and propos¬ 
als. One emphasis was 21st century skills, generally articulated as abilities to 
not only use technology, but to use it in ways analogous to modern work envi¬ 
ronments for information acquisition, analysis, and sharing. Grant announce¬ 
ments tended to cast this concern as a workforce-development issue (National 
Science Foundation 2007,2008). Proposals also stressed the cognitive benefits 
of ICT for engaging students in learning and invoking higher-order cognitive 
skills. In doing so, they cited a mix of research (e.g., Barak, 2004; Bransford, 
Brown, & Cocking, 1999), policy documents (e.g., Commission on Educa¬ 
tional Technology, 2005) and vendor claims (e.g., Creative Learning Systems, 
2011). The NETS fit in with this approach, contrasting traditional “teacher- 
directed, memory-focused instruction” and “knowledge from limited, authori¬ 
tative sources” with newer “student-centered, performance-focused learning” 
and “learner-constructed knowledge from multiple information sources and 
experiences” (ISTE, 2008, p. 4). 

In the dominant paradigm, new knowledge is not simply transmitted by 
a teacher. Rather, the projects deemed it important that students express and 
use their skills with hands-on activities in group settings that would make 
their knowledge manifest, shared, and open to revision and extension. This 
constructivist, Vygotskian notion that learning is social and takes place at 
the frontiers of an individual’s prior and new knowledge (Vygotsky, 1978) 
was assumed as often as it was articulated, but the selection of references and 
activities makes clear that it was present in all of the projects discussed here. 

Inevitably, there were differences across classrooms in the extent to which 
students were able to participate in hands-on, student-centered learning. 
These differences showed up on the ICOT observations as different propor¬ 
tions of time spent with technology by students and teachers, a preponder¬ 
ance of one or another teacher role, and observer ratings of more or less 
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essential uses of technology. In the original contexts of the observations, the 
presumptive reasons for these classroom differences had to do with varia¬ 
tions within the grant-funded professional development projects themselves. 
The numbers of teachers involved in most projects was small (less than 30). 
The identities and unique professional development histories of individu¬ 
als were well known. No attempt was made to generalize from each of these 
small samples beyond their respective projects because the local factors 
(e.g., being in a particular training cohort) were so obvious and because the 
numbers were so small. 

However, over a period of 2 years and numerous observations, a more 
general research question emerged. Given the common technology stan¬ 
dards and pedagogical values, why the differences in technology integra¬ 
tion? One of the obvious possibilities—differences in resources—was not 
sufficient. Evaluators noted that classrooms in wealthy districts with lots of 
technology and a history of professional development did not necessarily 
provide more hands-on ICT activities for students. For this study, the ques¬ 
tion was distilled as: What are the observed relationships between students’ 
technology use and the technologies and classroom environments that 
teachers arrange for them? 


Method 


Sample and Data Characteristics 

ISTE observed 109 teachers from seven technology projects under U.S. Depart¬ 
ment of Education and National Science Foundations grants in two states during 
2008-2010. After 2 years, the ICOT database included 189 records from six ob¬ 
servers. I removed observations conducted by trainee observers that were miss¬ 
ing key data or that had contradictory records, leaving 144 observations from 
three researchers who had undergone common training (16 observations from 
one observer on two programs, 26 from one observer on three programs, and 
104 from one observer on seven programs). I retained observations for 85 teach¬ 
ers, with one to seven observations per teacher (mean = 1.69, median=l). Half 
of the observations were in science classrooms, 16% were in language arts, 19% 
were in math, 8% were in social studies, and the remaining 6% were in various 
electives or technology. About 3% of observations were in primary grades (K-2), 
27% were in elementary (grades 3-5), 55% in middle school (grades 6-8), and 
about 14% were in high school (9-12). 

For this cross-project analysis, I removed individual identifying informa¬ 
tion, except for arbitrary dummy codes to identify multiple instances of the 
same teacher or the same observer. 

Observation Process 

ICOT version 1.0 was used for all of these observations. Version 1.0, which 
included the original NETS for Teachers (ISTE 2002), was a free-standing 
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application written for the Adobe AIR platform. (In 2010, the ICOT was re¬ 
vised to emphasize the second edition of the NETS for Students [ISTE, 2007], 
and the application was moved to a macro-enabled Microsoft Excel spread¬ 
sheet.) Table 1 (p. 210) shows the full list of variables included in ICOT vl.0. 

ICOT observations typically last for the major part of a class period. 
ICOT observers try to be in classrooms at or near the start of a class 
period. Along with initial observations about the setting (number of stu¬ 
dents, presence of technology, room arrangement, special characteristics 
of the environment), the observer records a start time. Every 3 minutes 
after that, the observer checks boxes to indicate if students and/or teach¬ 
ers are using technology and whether they are using it for learning (a 
computer that is powered up but unattended does not count as “in use,” 
and looking for lost passwords or playing recreational games does not 
count as “in use for learning”). At the end of an observation, the ICOT 
calculates the total minutes observed, the proportions of that time that 
teachers and students used technology, and the proportions of technolo¬ 
gy-use time devoted to learning. 

A typical structure for a classroom with multiple computing devices 
was for a teacher to introduce a topic in one of the teacher-directed roles 
(lecturing, modeling, or interactive direction). This could be the read¬ 
ing of a poem, a review of the periodic table, or a demonstration of a 
science probe. Then the students would be “turned loose” as individuals 
or groups to study or create, with the teacher in a facilitating role. This 
might include conducting research on the web or in a lab, analyzing data, 
or producing a presentation. 

Student engagement (anecdotally cited as a benefit of technology by 
many of the observed teachers) is assessed by observing student behav¬ 
iors in relation to the 3-minute chart. A student distracted from the les¬ 
son for any reason over more than a single 3-minute period is counted as 
not engaged. The proportion of students left after subtracting disengaged 
students is the percent engaged. In most classrooms, engagement as mea¬ 
sured by this criterion was quite high (M = .93, SD = .09, or 1-2 students 
disengaged out of an average-sized classroom of 24 in this sample). 

All ICOT variables except one involve only “present/absent” decisions 
by the observer. The exception is a prompt for evaluators to rate the unique 
contribution of technology in comparison to alternatives. Essential indi¬ 
cates that the lesson could not have been conducted without the technol¬ 
ogy. A simulation that analyzes real-time data might be an example. Useful 
indicates that the integrated technology provided distinct advantages 
over conducting the lesson without the tools. Completing a research and 
writing project online in two periods that otherwise take a week would 
probably have this rating. Somewhat useful indicates that the technologi¬ 
cal approach was comparable but not superior to an alternative. Presenting 
similar content in a slideshow that on a chalkboard or overhead projector 
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Table 1 . ICOT Observation Variables 


Teacher Roles 

Lecturing 

Number of Students 

Interactive Direction 

Student Groupings 

Facilitate/Coaching 

Individual 

Modeling 

Pairs or Small Groups 

Moderate Discussion 

Whole Class 

Learning Activities 

Give Presentation 

Engaged % 

Create Presentation 

Technology Use Time 

Run Simulations 

Total minutes observed 

Research 

% of total minutes during which teachers used technology 

Information Analysis 

% of teacher technology use devoted to teaching and learning 

Write 

% of total minutes during which students used technology 

Take Tests 

Drill & Practice 

% of student technology use devoted to teaching and learning 

Hands-On Skills 

Need for Technology 

Not useful 

Technologies Used by Teachers (t) 

Somewhat useful 

and by Students (s) 

Useful 

Digital camera 

Wiki (s*) 

Essential 

Presentation 

NETS for Teachers (1 sl edition) 

Tablet computer 

1 A.1. operating system procedures* 

Document camera 

1 A.2. routine hardware and software problems 

Graphics (t*) 

1 A.3. content-specific tools 

Handheld 

1A.4. productivity tools* 

Desktop 

1A.5. multimedia tools* 

Word processing (t*s*) 

1 A.6. interactive communication tools 

Spreadsheets 

1A.7. curriculum-based presentations/publications 

Database (t*s*) 

1A.8. curriculum-based collaborations* 

Video production 

1 A.9. appropriate technology selected 

Concept map 

2A.1. developmental^ appropriate learning activities 

Interactive board 

2A.2. technology-enhanced instructional strategies 

Library database 

3A.1. learning experiences address content standards 

Laptop computer 

3A.2. learning experiences address student technology standards 

Clickers 

3B.1. technology supports learner-centered strategies 

Web browser 

3C.1. technology applied to develop students’ higher order skills 

Videoconferencing 

3C.2. teacher applies technology to develop students' creativity 

Email 

3D.1. class management facilitates engagement with technology 

Blog 

3D.2. technology integrated as a teacher tool 

Calculator 

3D.3. technology integrated as a student tool 

Simulation (s*) 

3D.4. student grouping varied as needed to facilitate learning 

Science probe 

4A.1. student learning of subject matter assessed with technology* 

Podcast 

4A.2. teacher assesses student technology skills 

Web authoring 

4A.3. teacher employs a variety of assessment strategies 

Video camera 

6A.1. teacher models legal and ethical technology practices 

Drill & practice 

6A.2. teacher explicitly teaches legal and ethical technology practices 
6B.1. diverse learners enabled and empowered.* 

6D.1. safe and healthy use of technology promoted* 

6E.1. equitable access to technology for all students. 


* Indicates significant differences in frequencies reporting by different observers of the same teachers. 
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Table 2, Observer Identifications of Technologies and Standards 


Observer 

# Teacher Technologies 

# Student Technologies 

# NETS Addressed 

1 (/V = 16) 

4.1 

3.8 

16.1 

2 (/V - 10) 

3.6 

3.3 

21.1 

3 (N- 29) 

1.6 

2.0 

10.6 


would be an example. Not useful indicates that the lesson would have ben- 
efitted from different media altogether. One example would be projecting 
a complex relationship (e.g., a complicated chemical reaction or a multistep 
math problem) piece by piece, one computer screen at a time, when the 
entire system could only be illustrated in a larger physical format such as 
a wall-sized chalk or dry-erase board. 

Observer Reliability 

The three observers had trained together on an earlier version of the 
protocol using the iterative procedure described by Kelly and Haber 
(2006). I computed chi-square frequencies on ratings for 12 teachers, 
whom were observed by each of the three observers on a total of 36 
occasions. I found no significant differences (p < .05) in the frequen¬ 
cies of most variables (i.e., in most cases, no observer appeared to be 
systematically stricter or more lax in coding an attribute as present). A 
comparison of mean estimates of student engagement also showed no 
significant differences across observers. I observed significant variation 
in coding for five technologies: databases, graphics, simulations, wikis, 
and word processing (flagged with asterisks in Table 1). Difficulties in 
coding were related to the converging nature of the technologies. For in¬ 
stance, “word processing,” once synonymous with a dedicated computer 
program, is now a common function within many other applications. 
Some observers recorded its presence only when a dedicated program 
was running; others checked the technology whenever any text editing 
function was in use. 

Proportions were also significantly different across observers for seven 
of the NETS*T indicators (flagged with asterisks in Table 1). To some extent, 
ambiguity in the NETS reflects the previous convergence of technologies 
(e.g., “productivity tools” [NETS’T 1A4] and “multimedia tools” [NETS*T 
1A5] may now reside within the same computer application. In other cases, 
the variability in recorded NETS and technologies may accurately portray 
what happened in the classroom of the same teacher on different occasions. 
However, the three observers recorded significant differences in the mean 
overall numbers of technologies and standards on the two programs they 
all observed. The differences across observers presented in Table 2 are sig¬ 
nificant (F[ 1,2] = 9.30 ,p <. 0001). 

In any case, conclusions based on the standards and technologies flagged 
in Table 1 need to be treated with caution. 
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Table 3. Teacher Roles and Learning Activities by Type 


Variable 

Levels of Variable 

Student-Teacher Locus 

Teacher rales 

Facilitation 

Moderated Discussion 

Lecture 

Student-Centered 


Interactive Direction 

Teacher-Centered 

Student learning activities 

Modeling 

Creating Presentations 

Presenting 

Writing 

Information Analysis 

Research 

Student-Directed (study, creation, and communication) 


Simulations 

Drill And Practice 

Flands-On Training 

Teacher-Directed (instruction and assessment) 


Taking Tests 



Analysis 

The analysis looked at seven types of variables within the ICOT: (a) teacher 
roles, (b) student groupings, (c) student learning activities, (d) the amount 
of time the technology was used, (e) the types of technology used, (f) 
student engagement, and (g) the need for technology use. The projects 
for which these observations were conducted were intended to influence 
these variables in particular ways. They were designed to promote: 

• Teachers in roles as facilitators, modelers, and moderators 

• Student collaborative work 

• Student creativity and communication within the curriculum 

• Student interaction with technology 

• Student engagement in class 

A first step was to see if there were patterns of teacher roles, student 
groupings, and learning activities that were related to different levels of stu¬ 
dent engagement with technology. The initial finding was that, at this level 
of detail, almost every observation was unique. The ICOT has a total of 17 
levels across these three variable types. In 144 observations, there were 127 
unique patterns of these 17 attributes. Clearly some collapsing of categories 
would be necessary to find meaningful patterns across observations. 

The constructivist theory behind the programs for which the observers 
collected these data emphasized increasing hands-on, creative, student- 
directed activities versus teacher-directed, knowledge-transfer pedagogy. In 
terms of locus on this student-teacher axis, the ICOT includes two student- 
centered teacher roles (facilitation and moderation) and three teacher- 
centered roles (lecture, interactive direction, and modeling). Within student 
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activities, the ICOT includes six that are primarily student-centered creation 
and study (creating/delivering presentations, writing, research, information 
analysis, and running interactive simulations), and three that are primarily 
teacher directed (tests, drill and practice, and hands-on skill training), as can 
be seen in Table 3. 

When collapsing classroom characteristics in terms of student/teacher cen- 
teredness and direction, the data set had 28 unique patterns for 116 observa¬ 
tions for which all these data points were filled in, as shown in Table 4 (p. 214). 

With the levels collapsed, 10 patterns of teacher roles, student groupings, 
and learning activities can describe three-quarters of the observations. Most 
patterns show a mix of teacher-centered and student-centered teacher roles. 
Student-directed activities appeared twice as often as did teacher-directed 
activities. Whole-class instruction was observed in most periods, but in 83% 
of cases, it was mixed with individual and small-group work. Teacher roles 
also tended to be mixed in a period, with 70% of observations noting both 
teacher-centered and student-centered roles. Activity types tended to be 
more homogenous within a period, with 78% of observations noting either 
student-directed or teacher-directed activities. 

It is important to note that all these attributes of the classroom are not 
necessarily recorded at the same time. For example, when we look at the 
correlations of roles, groupings, and activities, we find that there is no cor¬ 
relation between teacher-directed activities and whole-class organization 
(Table 5, p. 215). That does not mean that teachers generally conduct class 
as a whole group without directing learning activities (although that can 
occur). Rather, it means that the fact that teachers convene the class as a 
whole during some part of a period has no relation to whether they include 
student-directed learning activities in some part of the same period. On the 
other hand, there are only so many minutes in an hour, and teachers make 
decisions about how to spend that time. As noted above, types of learning 
activities tend to be mutually exclusive. 

A certain mix of differing roles and groupings appears to be common 
and possibly essential at certain points in a lesson or unit. An approach 
that many teachers in these technology initiatives followed was to begin 
a lesson with lecture and modeling of the technology procedures to 
be used, after which students were “turned loose” as individuals or in 
groups to study a topic, take assessments, or create products. These 
scenarios would be reflected in patterns such as 1,2, 3, and 5. Simpler 
patterns were generally “steady-state” periods in which a particular activ¬ 
ity dominated. An observer coming into class that had begun a project 
in a previous period might see students enter a room and independently 
return to their ongoing work. The observer would describe that environ¬ 
ment in a manner similar to pattern 15 and would describe a period in 
which students take notes on a lecture or undergo an assessment in a 
manner similar to pattern 13. 
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Table 4. Observed Group/Role/Activity Patterns by Frequency 


Pattern 


Student Groupings Teacher Roles Learning Activities 


Pattern 

# 

Individual 

Work 

Pairs/ 

Small 

Groups 

Whole 

Class 

Student- 

Centered 

Teacher- 

Centered 

Student- 

Directed 

Teacher- 

Directed 

Frequency 

% 

(N =16) 

Cumulative 

% 

i 


X 

X 

X 

X 

X 


15 

0.13 

0.13 

2 

X 


X 

X 

X 

X 


14 

0.12 

0.25 

3 

X 


X 

X 

X 

X 

X 

12 

0.10 

0.35 

4 



X 

X 

X 

X 


9 

0.08 

0.43 

5 

X 


X 

X 

X 

X 

X 

8 

0.07 

0.50 

6 

X 



X 


X 


7 

0.06 

0.56 

7 


X 

X 

X 

X 


X 

7 

0.06 

0.62 

8 

X 



X 

X 

X 


5 

0.04 

0.66 

9 



X 


X 

X 


5 

0.04 

0.71 

10 

X 

X 

X 


X 

X 


5 

0.04 

0.75 

11 


X 


X 

X 

X 


3 

0.03 

0.78 

12 



X 


X 


X 

3 

0.03 

0.80 

13 

X 


X 


X 


X 

3 

0.03 

0.83 

14 

X 



X 

X 


X 

2 

0.02 

0.84 

15 


X 


X 


X 


2 

0.02 

0.86 

16 


X 


X 


X 

X 

2 

0.02 

0.88 

17 


X 



X 


X 

2 

0.02 

0.90 

18 



X 

X 

X 


X 

2 

0.02 

0.91 

19 

X 



X 



X 

1 

0.01 

0.92 

20 


X 


X 

X 

X 

X 

1 

0.01 

0.93 

21 



X 

X 


X 

X 

1 

0.01 

0.94 

22 

X 

X 


X 

X 

X 


1 

0.01 

0.95 

23 

X 

X 


X 

X 


X 

1 

0.01 

0.96 

24 

X 


X 


X 

X 


1 

0.01 

0.97 

25 


X 

X 

X 


X 


1 

0.01 

0.97 

26 


X 

X 

X 



X 

1 

0.01 

0.98 

27 

X 

X 

X 


X 


X 

1 

0.01 

0.99 

28 

X 

X 

X 

X 

X 

X 

X 

1 

0.01 

1.00 

Relative 

Frequency 

0.53 

0.37 

0.77 

0.83 

0.87 

0.80 

0.41 





by Level 


Patterns of Technology Use 

The amounts of time that technology is used by students and teachers are 
related to individual and whole class groupings, student-centered teacher 
roles, and student study activities (see Table 6). In the case of groupings and 
roles, the relation between teacher and student technology use is reciprocal. 
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Table 5. Correlations between Groupings, Roles, Technology Use and Need, and Student Engagement 





Grouping 


Teacher Roles 

Learning Activities 






Student- Teacher- 

Student- 

Teacher- 



Individual 

Groups 

Whole Class 

Centered Centered 

Directed 

Directed 

Individual 


1.000 







Groups 


-0.4434 

1.000 






Whole Class 


-0.6694 

0.4604 

1.000 





Student-Centered 


0.106 

0.2374 

-0.161 

1.000 




Teacher-Centered 


0.090 

0.099 

0.112 

-0.257T 1.000 



Student-Directed 


0.196* 

0.090 

-0.083 

0.211* 

0.038 

1.000 


Teacher-Directed 


-0.112 

0.117 

0.001 

0.069 

0.047 

-0.4544 

1.000 

Correlation significant, p < .05. 
t Correlation significant, p <.01. 
tCorrelation significant, p < .001 








Table 6. Correlations between Groupings, Roles, Technology Use and Need, and Student Engagement 




Grouping 

Teacher Roles 

Learning Activities 

Technology 

Use 


Need for 








% 


Small 

Whole Student- 

Teacher- 

Student- 

Teacher- 


Students 

Tech in 

Individual 

Groups 

Class Centered 

Centered 

Directed 

Directed 

Student Teacher Engaged 

Lesson 

Student 

Tech 0.2664 

Use 

0.125 

0.3454 0.3504 

0.013 

0.167 

0.161 

1.000 



Teacher 









Tech -0.3484 

Use 

0.143 

0.2934 0.4424 

0.053 

0.3144 

0.111 

-0.4414 1-000 



% Stu- 









dents 0.119 

0.089 

0.119 0.148 

0.044 

0.027 

0.010 

0.216* -0.199* 1.000 


Engaged 

Need for 









Tech in 0.3004 
Lesson 

0.059 

0.3004 0.280T 

0.020 

0.3404 

0.013 

0.5144 0.3394 0.154 

1.000 


Correlation significant, p < .05. 
t Correlation significant, p <.01. 
t Correlation significant, p <.001. 


Observers saw relatively more student technology use and less teacher tech¬ 
nology use in classrooms with an emphasis on individual student work and 
student-centered teacher roles. This same pattern was observed for small- 
group classroom organization, but the difference was not significant. A lower 
proportion of time with student technology use and a higher proportion of 
teacher use were observed in classrooms with whole-class organization. 

Student and teacher technology use are negatively correlated. Student 
study activities were associated with more time spent in teacher and student 
technology use. 
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Table 7. Correlations of Classroom Characteristics with Technologies That Students (S) and Teachers (T) Use Frequently 


Technology 

User, % 
(N= 134) 


Grouping 


Teacher Roles 

Learning Activities 

Technology Use 

% 

Students 

Engaged 

Need for 

Tech in 

Lesson 

Individual 

Small 

Groups 

Whole 

Class 

Student- 

Centered 

Teacher- 

Centered 

Student- 

Directed 

Teacher- 

Directed 

Student 

Teacher 

Presentation 

T 24% 

-0.018 

0.132 

0.163 

-0.037 

0.052 

0.131 

-0.044 

-0.040 

0.040 

-0.148 

0.223* 

Software 

S10% 

0.041 

-0.017 

-0.050 

0.068 

0.056 

0.249f 

-0.225* 

0.116 

-0.103 

0.112 

0.150 

Interactive 

T 48% 

-0.197* 

-0.150 

0.260f 

-0.258f 

0.063 

-0.341$ 

0.111 

-0.350$ 

0.684$ 

-0.178* 

-0.338$ 

Whiteboard 

S 25% 

-0.237f 

-0.107 

0.151 

-0.255f 

0.019 

-0.428$ 

0.246$ 

-0.296$ 

0.569$ 

-0.116 

-0.251$ 

Response 

T10% 

-0.085 

0.012 

0.076 

-0.209* 

0.069 

-0.366$ 

0.325$ 

0.147 

0.306$ 

-0.025 

-0.119 

System 

S10% 

-0.055 

-0.010 

0.044 

-0.122 

0.081 

-0.396$ 

0.400$ 

0.174* 

0.349$ 

0.019 

0.147 

Web 

T 25% 

0.214* 

0.170 

-0.054 

0.156 

0.201* 

0.432$ 

0.003 

0.268$ 

-0.217* 

-0.031 

0.433$ 

Browser 

S 34% 

0.301$ 

0.065 

-0.227f 

0.191* 

0.148 

0.556$ 

-0.211* 

0.369$ 

-0.425$ 

-0.048 

0.422$ 


Correlation significant, p < .05. 
t Correlation significant, p <.01. 
t Correlation significant, p <.001. 


Technology Need and Technology Use 

Observers’ ratings of the need for technology within a lesson approximated 
the size and direction of the various correlations with proportion of time in 
which students used technology. In classes where students were observed 
to use technology more, work as individuals, work with teachers who adopt 
student-centered roles, and engage in study activities, observers tended to 
rate the technology use as more essential to the lesson. In classrooms that 
emphasized whole-class instruction with the teacher using technology for 
a larger proportion of time, observers tended to rate technology use as less 
necessary to learning. 

Student Engagement and Technology Use 

There is an inverse relationship between the amount of time teachers and stu¬ 
dents use technology. Further, observers tended to rate engagement higher as 
student technology-use time increased and lower as teacher technology-use time 
increased. To make this more concrete, Table 7 lists the most common hardware 
and software—those that observers saw teachers and students using in at least 
10% of classrooms—along with the correlations between each technology’s use 
and other classroom characteristics (observers also noted frequent use of word 
processing and simulations, but because of previously mentioned concerns 
about coding, Table 7 does not include those applications). 

The strongest positive relationships were between interactive whiteboards 
and teacher technology use (i.e., teachers with whiteboards use them a lot). 
The strongest negative relationships were between student Web browsing 
and teacher technology use (those tend not to occur together in the same 
period) and between student whiteboard use and student-directed learning 
(whiteboards tend to be used for teacher-directed learning). Other strong 
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Table 8. NETS-T (2000) Performance Indicators, % of Observations in Which Addressed (/V = 144) 


Standard 

Performance Indicator 

% 


1 A.1. operating system procedures* 

0.30 


1 A.2. routine hardware and software problems 

0.47 


1 A.3. content-specific tools 

0.42 


1 A.4. productivity tools* 

0.35 

Technology Operations & Concepts 

1A.5. multimedia tools* 

0.23 


1 A.6. interactive communication tools 

0.10 


1A.7. curriculum-based presentations/publications 

0.58 


1A.8. curriculum-based collaborations* 

0.13 


1 A.9. appropriate technology selected 

0.83 


2A.1. developmental^ appropriate learning activities 

0.83 

Designing Learning Experiences 

2A.2. technology-enhanced instructional strategies 

0.81 


3A.1. learning experiences address content standards 

0.84 


3A.2. learning experiences address student technology standards 

0.58 


3B.1. technology supports learner-centered strategies 

0.53 


3C.1. technology applied to develop students’ higher-order skills 

0.31 

Teaching, Learning, & Curriculum 

3C.2. teacher applies technology to develop students’ creativity 

0.15 


3D.1. class management facilitates engagement with technology 

0.63 


3D.2. technology integrated as a teacher tool 

0.64 


3D.3. technology integrated as a student tool 

0.58 


3D.4. student grouping varied as needed to facilitate learning 

0.38 


4A.1. student learning of subject matter assessed with technology* 

0.22 

Assessment & 

Evaluation 

4A.2. teacher assesses student technology skills 

0.03 


4A.3. teacher employs a variety of assessment strategies 

0.06 


6A.1. teacher models legal and ethical technology practices 

0.00 


6A.2. teacher explicitly teaches legal and ethical technology practices 

0.03 

Social, Ethical, Legal, Human Issues 

6B.1. diverse learners enabled and empowered* 

0.00 


6D.1. safe and healthy use of technology promoted* 

0.09 


6E.1. equitable access to technology for all students 

0.54 


associations are between Web browsing and lessons in which observers rated 
technology use as valuable, and between Web browsing and lessons that 
involved student-directed learning. 

NETS Teacher Standards 

Table 8 shows the percentage of observations in which observers recorded 
each of the performance indicators from Table 1. Recall that seven indicators 
(1A.1, 1A.4, 1A.5, 1A.8, 4A.1, 6B.1, 6D.1) had inconsistent recording across 
reviewers; those percentages may not be accurate. 

Leaving out the seven suspect items, the mean number of performance 
indicators observed per observation was 9.33 (range 0-19, standard devia¬ 
tion 4.62). 
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Table 9. Correlations of Numbers of Observed NETS*T Indicators with Classroom Attributes 




Grouping 


Teacher Roles 

Learning Activities 

Technology Use 

Need for 


Individual 

Small 

Groups 

Whole 

Class 

Student- 

Centered 

Teacher- 

Centered 

Student- 

Directed 

Teacher- 

Directed 

Student 

Teacher 

Tech in 

Lesson 

r 

0.182* 

0.134 

-0.177* 

0.306t 

0.134 

0.3611 

-0.027 

0.412t 

-0.233t 

0.476L 

N 

142 

142 

142 

131 

131 

128 

128 

144 

144 

138 


* Correlation significant, p < .05. 
t Correlation significant, pc. 01. 
tCorrelation significant, p < .001. 


The NETS observations suggest that these technology-using classes 
displayed intentional design of instruction that integrated technology 
and content. Observed classroom management and student access were 
not somewhat lower than the observed evidence of preparation, sug¬ 
gesting that executing a lesson can be harder than planning it. Less- 
than-100% observations of “equitable access” do not mean that teachers 
treated students inequitably, but rather reflect the fact that some of the 
observed schools had technology infrastructure or support issues that 
constrained teachers’ lessons. The frequency of observed “learner-cen¬ 
tered strategies” is relatively high in terms of other NETS indicators, but 
lower than the recorded frequencies of student-centered activities (the 
correlation between the two variables was .36). Although some of these 
instances may be recording inconsistencies, they also reflect situations 
where learner-centered activities in a classroom did not make effective 
use of technology. 

In terms of other classroom attributes, frequency of observing the NETS*T 
was positively correlated with individual student work, student-centered 
teacher roles, student-directed activities, a large proportion of student technol¬ 
ogy use, and observers’ judgments about the need for technology. The fre¬ 
quency of observing the standards was negatively associated with whole-class 
grouping and higher levels of teacher technology use (see Table 9). 

In terms of commonly used applications, the number of observed NETS'T 
indicators was positively associated with student use of presentation software 
and student or teacher browsing of the Web. The number of observed indica¬ 
tors was negatively associated with interactive whiteboard use (see Table 10). 

Discussion 

These results warrant three areas of discussion: interpretation of the correla¬ 
tions, the observation process, and the use of the NETS in evaluation. 

Interpreting Results in Context 

The question behind this investigation had to do with the relationships 
between classroom attributes, so interpretation of the correlation coefficients 
is the crux of the study. Perhaps the most important thing to keep in mind 
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Table 10. Correlations of Numbers of Observed NETS*T Indicators with Common Applications 


Technology 

User 

r 


T* 

0.118 

Presentation Software 


S* 

0.286T 


T 

-0.285t 

Interactive Whiteboard 


S 

-0.237f 


T 

0.012 

Response System 


S 

0.001 


T 

0.434± 

Web Browser 


S 

0.472T 


*T= Teachers, S = Students 
t Correlation significant, p <.01. 
t Correlation significant, p <.001. 


is the context of the observations. The data on interactive whiteboards are 
illustrative. Teachers told observers they were very appreciative of interac¬ 
tive whiteboards. One of the programs the observers were evaluating was 
devoted to providing the boards and associated training throughout several 
school districts. As might be expected, given that the boards are a presenta¬ 
tion medium, they are positively associated with whole-class instruction by 
the teacher. They are negatively associated with student-centered, individual, 
student-directed activities and with student use of technology in general. 
Typical use involved teacher lectures and demonstrations, although some 
classes used them frequently for student presentation of work. Some of the 
whiteboards’ special features (hiding/showing images and integration with 
student response systems) made them effective as formative assessment 
tools. Teachers who used the boards invariably cited greater student engage¬ 
ment with material on the whiteboard compared to content presented with 
conventional lectures and visual aids. 

Why then was there a negative correlation with engagement in Table 
7? In classrooms with interactive boards, there may be no alternative. The 
board is often mounted on top of an older dry-erase or chalkboard. Once 
in place, it is used for everything from interactive simulations to the daily 
lunch menu. Observed engagement with effective lessons was sometimes 
offset by observed distraction during mundane activities that used the same 
technology. Thus observers often rated the board as only “somewhat useful: 
other approaches would work as well.” That was true, although the obvious 
alternative (the chalkboard) was no longer accessible. 

On the other hand, Table 7 does accurately represent the facts that teach¬ 
ers primarily used whiteboards and that high levels of teacher technology 
use were associated with lower levels of student engagement (see Table 6). 
Engagement was generally higher in dispersed-technology classrooms (labs 
or conventional classrooms with laptops), where students were more likely 
to use the technology than were teachers. 
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This is not to say that technology necessarily dictates practice. Observ¬ 
ers encountered a few whiteboard-equipped classrooms where the teachers 
turned over operation of the technology to the students and then facilitated 
group use of these resources. This occurred in classrooms where teach¬ 
ers had at least one year of experience with the technology. Some of these 
instances involved interactive slates that allow the board to be operated from 
anywhere in the room. This tended to make the board more of a shared 
workspace than a forum for “stand-and-deliver” recitation at the board. 
These experiences suggest that flexibility in teaching with technology can be 
increased with experience and with more sophisticated tools. 

However, there seems to be a path of least resistance with any technol¬ 
ogy. Whiteboards and presentation software are designed for an individual 
or small team to deliver information to a group. Laptops and Web browsers 
were designed for use by individuals. Alternative uses require additional 
technology and/or more energy in terms of classroom management and 
lesson planning. Educational planners need to be aware of these pedagogical 
pressures and relate them to their own priorities. A technology implemented 
in response to one need (e.g., formative assessment by the teacher) may have 
unintended consequences, such as reduced time or technology available for 
mastering “21 st century skills” such as online research and collaboration. 

Observations of Technology-Using Classrooms 

The 2 years of observations provided several guidelines for observation 
practice. One is that the recording information electronically can greatly 
reduce total time for collecting and using observation data. The main sav¬ 
ings is in transcription, which, in the case of paper forms, can take as long as 
the original collection. The first version of the ICOT (used in the observa¬ 
tions presented here) had a cumbersome export function that did not take 
full advantage of this capability. Later versions corrected that problem. Any 
computer-based observation tool should have the ability to export files that 
can be opened in common statistics packages. The cost savings can make 
real-time observations a viable strategy in cases where they otherwise would 
be unfeasible. 

Another guideline is that observation programs should be prepared 
for the wide diversity in classrooms. Even among projects that have gross 
similarities in purpose, professional development, and technology, there 
are numerous patterns of classroom attributes (see Table 4). The ICOT was 
designed so that its component sections can be used independently. This 
might apply when an administrator does a walk-through of a building to get 
an idea of how many teachers are using technology at all, or when assuming 
a particular instructional role. The ICOT observations suggest caution in 
this approach. Most classrooms display a wide range of attributes in a single 
period. Observing a particular attribute during a short slice of time may 
not be a reliable gauge of practice. Even full-period observations need to be 
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informed by interviews or debriefs so that the context is clear. For example, 
slow transitions between activities that are picked up on the ICOT 3-min¬ 
ute chart as “technology used/not for learning” might be due to inadequate 
professional development or technical problems, or because an adventur¬ 
ous teacher is trying something new with full awareness that the lesson may 
encounter difficulties. 

That in turn may affect how an observer codes other attributes, such as 
the need for technology or the attainment of standards. This level of inter¬ 
pretation is best attained by a project team agreeing on its criteria and con¬ 
ducting joint training with multiple observers (Fish, Hayden, & Bielefeldt, 
2005). Even with this experience, the ICOT data presented in Table 2 suggest 
that observers need to periodically recalibrate their coding practices through 
joint observations and discussions. 

Observing the NETS 

As noted earlier, the NETS were the most difficult subsection of the ICOT to 
record consistently. Assessing the NETS has been a concern since the stan¬ 
dards were first published in 1998 and continues to be a work in progress 
with the revised standards. Various products and services claim to be or are 
explicitly designated as “NETS-aligned” for purposes of assessment, instruc¬ 
tion, or professional development (ISTE, 2011). 

If anything, the challenge has increased with the revision of the stan¬ 
dards that began in 2007. Whereas the original standards emphasized the 
use of the technology, the current standards emphasize the outcomes to be 
achieved with technology. “Teacher applies technology to develop students’ 
creativity” (NETS’T, first edition, indicator 3C.2) has become “Teachers ... 
promote, support, and model creative and innovative thinking and inven¬ 
tiveness” (ISTE, 2008). The revised NETS'S, which are used in the current 
version of the ICOT, elevate creativity and innovation to a standard in and of 
itself: “Students: (a) apply existing knowledge to generate new ideas, prod¬ 
ucts, or processes; (b) create original works as a means of personal or group 
expression; (c) use models and simulations to explore complex systems; and 
(d) identify trends and forecast possibilities” (ISTE, 2007). 

The student creativity standard involves the observer in multiple judg¬ 
ments, including whether the student really applies existing knowledge and 
whether that application actually results in the generation of new ideas. All 
these decisions come before even considering whether technology was used 
effectively in those pursuits. In any specific context, the NETS observer 
exploring this standard may be compelled to make the same type of deci¬ 
sions required in a study based on content frameworks such as the Common 
Core Standards for Mathematical Practice (Common Core State Standards 
Initiative, 2011). 

Ultimately, NETS-based evaluations will have to make links with content. 
Even if one believes that the NETS embody skills that are valuable in their 
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own right, we have to observe those skills in a specific context of teaching 
and learning. And the NETS are meaningful to the extent that they point to 
abilities beyond that context. One development that would contribute to im¬ 
proving observational studies of ICT in education would be a concordance 
of the revised NETS with emerging standards such as the Common Core, 
such that observations coded in one framework have meaning for the others. 
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